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Data Use in the School and Classroom: 

The Challenges of Implementing Data-Based Decision Making Inside Schools 1 

Christopher A. Thorn 



Introduction 

This paper examines problems school-level staff encounter when attempting to 
implement data-based decision-making reform efforts, specifically those focused on teaching and 
learning in the classroom. The paper also offers recommendations for professional development 
that address gaps in traditional principal and teacher training. 

Many schools and districts are exploring data-driven decision making as a solution for 
improving resource allocation and instructional program decisions. One of the most challenging 
problems policy makers and educators face in attempting to implement curriculum reforms is 
that intervention decisions are made at least one organizational level above that of the teachers — 
the persons actually engaged in instruction. 

Any systemic effort to implement a focus on data-based decision making at the school 
and classroom levels faces several challenges. First, most data available within district 
information systems are limited to what has been deemed important for the operational needs of 
schools and the district and are only available on systems supported by centralized computing 
services. These data include attendance, discipline, and basic demographic data. District systems 
also contain detailed information about human resources, budgets, and other business processes. 
Typically, the only available outcome data are grades and the results from centrally (and often 
annually) administered tests. These data, although useful to help frame annual analysis of 
school-, classroom-, or student-level outcomes, are inadequate for making midcourse or interim 
instructional decisions within a single grade or marking period. 

Second, educational organizations have much to leam about the complexities of 
integrating data effectively into their decision-making processes. The issues involved in 
successful data-based decision making are beginning to be discussed in the literature on 
educational administration and assessment, and an important and growing body of relevant work 
is emerging from business schools around the world. The business studies range from 
considerations of the role of experts in organizational learning (Albert & Bradley, 1997) to 
multidimensional representations of the life cycle of knowledge (Boisot, 1998). The work that is 
emerging from educational sources tends to focus on application. For example, the journal 
School Administrator recently published a special issue (April 2001) dedicated to data-driven 
decisions — in particular, the question of what data and what representational forms are 
appropriate for district- and school-level administrators as they attempt to evaluate the impact of 
curricular changes (Creighton, 2001). This work addresses systemic reform concerns that help to 
provide the technical and information resources necessary to support school-level data-based 
decision making. 



1 Presented at the 83 rd annual meeting of the American Education Research Association, April 1-5, 2002. 
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Third, although large-scale efforts to make assessment and accountability data more 
generally available do provide some insight into the performance of an educational system, it 
seems reasonable to assume that there are major differences between the evidence used for 
external accountability systems and the data needed for making instructional decisions on a 
quarterly, weekly, or daily basis in the classroom. The large gap between the time horizons of 
state-level testing and the more immediate needs of program administrators and individual 
teachers in the classroom means that data needs and rules of evidence will be commensurately 
divergent. Moreover, many teachers (and administrators) do not have the skills to adequately 
understand and accurately use standardized test results and accountability data. The problem at 
this level is ensuring that building-level leaders and individual teachers have the necessary skills 
to engage in action research, data analysis, and evaluation. 

Approaches to Understanding Decision Making and Knowledge Work 

A growing literature on information seeking, information processing, and information use 
provides insights into how individuals and groups identify information needs and then respond 
(or choose not to respond) to those needs (Choo, Detlor, & Turnbull, 2000; Davenport & Prusak, 
1998; Ford, 2001; Hoglund & Wilson, 2000, 2001; Wai-yi, 1998). This work draws on and can 
be used to frame other work in the areas of group decision making (Alter, 2002; Bhatt & Zaveri, 
2001; Eden & Ackermann, 1998; Fazlollahi & Vahidov, 2001; Wenger & Snyder, 2000), 
knowledge management (Creighton, 2001; Davenport & Marchand, 1999; Greengard, 1998; 
Harrison & Pelletier, 2001; Hibbard & Carrillo, 1998; Jasinski & Huff, 2002; Thorn, 2001), and 
the human factors of decision support systems (Albert & Bradley, 1997; Alter, 2002; Bhatt & 
Zaveri, 2001; Boisot, 1998; Eden & Ackermann, 1998; Harrison & Pelletier, 2001; Shneiderman, 
1998). 



One recent study of information seeking on the Web provides an excellent synopsis of 
what the authors call an integrated model of human information seeking (Choo et al., 2000; see 
Figure 1, below). The important aspects of the model for this paper are the intersections between 
the different behavioral areas — the identification of information needs, the search for the needed 
information, and the use of information to address the identified needs. The three challenges 
outlined in the introduction can be addressed through this model. To illuminate several aspects of 
this information- seeking model, the following discussion uses as an example a school 
improvement team attempting to create an improvement plan for math instruction. The anecdotes 
described in this case are taken from lessons learned from working with school improvement 
teams and from school improvement plan documents. 

Information Needs 

The identification of information needs is a primary goal in any type of school reform or 
curriculum improvement. As indicated above, much of the information available from district 
information systems is limited to data useful to make district-level decisions. The granularity 2 



2 Granularity is a term used to describe the level of aggregation of data. For example, attendance data could be listed 
as follows in increasingly fine granularity: days absent this year, days absent this semester, days absent this week, or 
periods absent this day. The finer the grain size, the more detailed the analysis can be. The tradeoff, however, is that 
the finer the granularity, the more data one must manage. 



2 




Data Use in the School and Classroom 



and temporal resolution' of the available data severely restrict its usefulness for different user 
groups. 




Figure 1. Human information seeking: An integrated model (Choo et al., 2000, p. 22). 

For example, in an effort to understand current performance and its relationship to school 
and district goals, a school improvement planning (SIP) team might examine district 
accountability reports for aggregate data and school- and individual-level score sheets from the 
math component of the annual standardized math achievement test, and then compare the results 
on these metrics to the performance goals set by the school, district, and/or state. In this situation, 
the SIP team would attempt to frame a problem. For example, fourth- and fifth-grade students 
seem to display declining scores on the math component of a district- wide test, leaving a gap 
between the observed performance of the students and outcomes expressed by the accountability 
goals. At this point, the problem has been defined as a gap. Typically, the data contained in the 
accountability system are not sufficient to diagnose the cause of the performance gap. 
Information on curriculum, teacher ability, classroom resources, attendance, discipline problems, 
and contributing social/home and other factors comes from many different sources. The next step 
is a decision-making process that produces one of two outcomes: Either the SIP team identifies 



3 Temporal resolution refers to the span of time to which a particular datum or data set refers. Annual test scores 
have the temporal resolution of one year. Weekly spelling test scores have the temporal resolution of one week. The 
temporal resolution of a particular type of data makes it more or less useful for measuring the state of or the change 
within a system over a given span of time. 
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gaps in their knowledge about what might explain the performance gap and initiates a search for 
additional information, or the team chooses to avoid the problem. Avoidance might take the form 
of a simple denial of the problem (attributing the gap to bad tests, high student turnover, etc.). 
Alternatively, the team might choose a solution based solely on the easily available aggregate 
data and make an intervention decision based on incomplete (or nonexistent) data. 

Information Seeking 

Information seeking is the most intriguing part of the information-processing model as 
applied to school-level decision making and the use of decision support tools. Once an 
information gap is identified, the SIP team must devise a model of learning that encompasses the 
outcome and includes factors that the team feels contribute — either directly or indirectly — to that 
outcome. It is at this point that the search for new sources of information begins. 4 This data may 
reside in teacher grade books, lesson-planning software, local databases, or locked filing 
cabinets. The team is faced with the task of assembling data that comply with the model of 
learning they have constructed. One important dimension of the information- seeking activity is 
relative ease of access. Some data will be readily available in a central data store or in the school 
office. Other data will be in paper form in or on an individual teacher’s desk. Resolving access 
issues is a constant struggle, since the resources needed to acquire hard-to-find or hard-to- 
manage data may exceed the data’s value in the analysis. 

Issues of the reliability and validity of the data are also important components of the 
discussion, particularly if students or teachers with substandard performance are to be exposed to 
serious consequences. In-class assessments, for example, may be entirely valid measures in that 
they accurately represent the content of the curriculum and related learning standards. The fact 
that they are scored by different teachers and that the sample sizes are small (individual 
classrooms) means that scores may not be reliable. It is at this point that tools for supporting data 
collection and data exploration are most important. The ability to visualize student outcomes 
across any of these data types is essential to show how consistently a certain group of students 
performs at a particular level. 

There are several districts in Wisconsin where district-level information technology (IT) 
and/or research units are attempting to make more and more data available through a data 
warehouse interface. Some of these efforts rely on (a) warehouse access for accountability and 
school management data and (b) local tools (spreadsheets, database packages, statistical analysis, 
report generators, etc.) to allow school-level staff to combine local information with district-level 
data for a variety of possible uses. Decision support models need to reflect both a district’s 
governance structure and its IT infrastructure. 

Information Use 

The information use portion of the integrated model of human information seeking (see 
Figure 1, above) describes the combination of the identified needs and the acquired information 



4 Some examples include attendance and tardiness data for individual students, discipline data, results of in-class 
assessments, seniority and educational level of teachers, quality of the curriculum, quality and availability of 
professional development in the area in question, etc. 
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and its translation into action. This is the point at which the gaps in student understanding or in 
teacher professional development are codified based on the gathered information and a plan is 
implemented to address the shortcoming. 

As the nodes within this section of the model indicate, information use interacts with 
human tendencies to maintain current practice and accept existing behavior norms. Information 
may also be used to choose new paths, reassign resources, and question existing rules. There are 
any number of instances in which school reform initiatives challenge existing practices, the 
allocation of resources, and the skills of teaching professionals. It is only in gathering appropriate 
information that newly defined problems can be examined and the results of new efforts can be 
evaluated. School reform efforts interact with all areas of this model. Policy makers and 
implementers must recognize these social structures and practices in order to support the desired 
behavior and achieve the target outcomes. 

Decision-Making Models 

Choo et al. (2000) provide a second model (see Figure 2, below) that parallels the human 
information-seeking model (see Figure 1, above). This model provides a shorthand for a larger 
decision-making model. This model has three parts: “sensemaking,” knowledge creation, and 
decision making. The authors argue that sensemaking is the first important activity. This is the 
process of coordinating beliefs with extant information. The sensemaking process often requires 
that one engage in knowledge creation — bringing external information together with implicit and 
explicit knowledge about the current problem. Knowledge creation serves to fill gaps in the 
sensemaking process — that is, it fills in gaps in understanding. Finally, one moves on to the 
decision-making process that integrates new understanding with existing rules and procedures 
that guide action. 

Knowledge-based systems are designed to support a decision-making model such as the 
one outlined above. Knowledge systems allow users to explore many alternatives in a series of 
“what if’ models. Systems that support exploration of large, complex data sets also support 
users’ development of increasingly sophisticated mental models. Use of the environment itself 
can contribute to the user’s personal skills as an analyst (Jasinski & Huff, 2002). School reform 
efforts encourage school-level staff to make informed decisions that both operationalize their 
own long-term strategies for student learning and professional growth and align with larger 
district and societal goals. 

Decision support systems can help to make sense of the overwhelming complexity of 
large data collections. They also can support users as they search for and explore the attributes of 
alternative strategies. The search for alternatives is one of the most important activities in which 
a SIP team engages. Shortcomings of existing outcomes lead to an identification of gaps, 
failures, and needs. The search for solutions is much enhanced if the governance structures and 
the norms of interaction support searching for alternative approaches. In situations in which the 
resources are fixed in a particular array — funding, staffing, curricular support materials, etc. — 
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alternatives may be quite limited. It is, however, the search for alternatives that provides a 
context for meaningful dialogue between professionals on a SIP team . 5 







Figure 2. Sensemaking, knowledge creating, and decision making (adapted from Choo et al., 2000, p. 64). 
Sensemaking 

Returning to our SIP team example, sensemaking is one of the core activities that is often 
overlooked in school improvement planning. Planning templates are usually created at the 
district level as a part of a district’s accountability system. Outcome metrics such as annual 
scores or percentiles apply a single accountability model to all schools equally. The sensemaking 
process provides the framework for local decision making. For example, when faced by 
declining math scores, a SIP team might look at the subscale test scores for both grade cohorts 
and individual students to see if they observe score declines across the board or in specific areas. 
This approach would allow the team to reconcile their local beliefs about the quality of student 
work and the relationship of existing classroom practices to that work. This process mirrors the 
information needs process in that it is through the identification of what is to be studied that 
information needs emerge. 



5 For a more technical discussion of methods used to generate alternatives, see Fazlollahi and Vahidov (2001). 
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Knowledge Creating 

The knowledge creation process can again be seen as parallel to the information-seeking 
process. Knowledge is created by assembling individual understanding; bringing in outside 
information in the form of possible alternatives, lessons learned, and the like; and relating these 
to information already in use. In our example of the SIP team, this process is likely to take the 
form of exploring what the math test actually measures. This information would then be 
combined with an understanding of the relevant teaching and learning standards to compare what 
is measured with what is being taught. This synthesis would then frame the gaps in the test and 
the gaps in the curriculum. This gap analysis combined with the sensemaking process would then 
provide a framework for crafting outcomes. 

Decision Making 

Once necessary new knowledge has been created and the SIP team has a clear 
understanding of students’ instructional needs and the areas that the test does not measure, it is 
time to examine a series of alternatives that would achieve the learning goal. This is another area 
in which teams are likely to experience great frustration. Often, schools have few resources for 
identifying alternative approaches to instructional challenges. They often have to rely on a few 
professional development days, personal experience, and limited contacts with other 
professionals outside of the school year. However, the team that takes the sensemaking process 
seriously — perhaps by neglecting the narrowly focused SIP framework — is often better prepared 
to make a more concrete statement and choose a plan that fulfills their understanding of the 
needs in the current situation. 

These models are not presented as a panacea for decision-making challenges, but they do 
provide a more human and responsible framework for understanding the dynamics of successful 
improvement planning at the school level. Effective decision support systems can make a critical 
difference at the decision-making stage of the SIP process. 

Characteristics of Group Decision Making 

Carneiro describes the group decision-making process in very accessible language 
(Cameiro, 2001, p. 223). He states that the process begins with a needs assessment and a 
determination of the scope of the problem. A decision team then moves on to collect 
information, evaluate alternatives, and implement the solution. 

Identification of individual preferences is vitally important to understand the outcomes of 
decision-making exercises. The social combination approach to decision making focuses on how 
much support each alternative has at the outset — before any discussion gets under way (Baron, 
Kerr, & Miller, 1992, p. 94). A school in which the majority of SIP team members want school 
improvement to focus on personal professional growth would take a very different approach than 
a school in which SIP members value collective action. The preferences of the team members 
participating in the decision-making process have a great deal of influence over the outcome, 
independent of any evidence that is introduced. This approach to understanding group decision 
making can be contrasted with the more traditional approach, which focuses on the content of 
group discussion to examine the movement from disagreement to consensus (Baron et al., 1992, 
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p. 104). It is the interplay between information brought to the group process and participants’ 
individual preferences that must be studied if one is to understand the decision outcomes. 

Strategic Decision Success 

Making successful strategic decisions has been studied a number of times. One 
interesting analysis of strategic choices provides a simple model that describes the challenges 
faced by decision makers in different organizational environments (Harrison & Pelletier, 2001). 
Harrison and Pelletier illustrate this point with a simple 2x2 matrix (see Table 1, below) that 
describes four different strategic decision-making outcomes (Types A-D). 

In this model, the rows represent the decision maker’s attitude toward a particular 
decision. Some actors are categorized as focusing on maximizing the outcome from a decision. 
As maximizers, decision makers in this group use a computational approach toward a decision, 
weighing the costs and benefits in an objective manner. Seeking to maximize outcomes also 
presumes that one has better information and more flexible resources. The other approach on this 
axis is that of seeking a satisfactory outcome. In this case, the decision maker seeks simply to 
meet the strategic goal. There is an acknowledgment that the decision maker is making a 
judgment call based on the best available data but that the focus is on achieving the goal, not 
maximizing the outcome. 

The columns in this model represent the decision maker’s attitude toward the decision- 
making process itself. Attitude is expressed in terms of the relative openness of the decision- 
making process and the attainability of the strategic goals. 

Harrison and Pelletier (2001) suggest that whereas Type D is a position that guarantees 
failure, Types A and C hold out some possibility of a successful outcome if the decision maker 
can see the incentive structure that suggests a change in attitude. 



Table 1 

Strategic Decision Matrix (adapted from Harrison and Pelletier, 2001, pp. 172-173) 





Attitude toward the decision-making process 


Attitude toward the decision 


Attainable objectives/open 
decision making process 


Unattainable objectives/ 
closed decision making process 


Judgmental decision making 
strategy/satisfying outcome 


Type A 


Type B 


Computational decision making 
strategy/maximizing outcome 


Type C 


Type D 



What this model means in a school decision support setting is that district- or school-level 
processes that do not allow teachers and other school-level practitioners to participate in the 
goals setting, needs assessment, implementation, and evaluation processes are very likely to fail 
because they do not include the appropriate participants. This lack of participation also increases 
the likelihood that the objectives will be unrealistic and unattainable. Along the same lines, a 
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strategy that focuses on extracting the maximum benefit from a particular intervention is 
guaranteed to rob much needed resources from other areas of the educational enterprise where 
they are needed just as desperately. 

Bhaat and Zaveri (2001) provide an alternative perspective on the characteristics of 
decision support systems. According to this model, decision support systems include a wide 
range of data management and data visualization features. They describe efficient access to data, 
the ability to experiment with variables to find new correlations, generation of alternative 
models, trend analysis of historical data, and summative evaluation as vitally important aspects 
of decision support if one is serious about promoting organizational learning (Bhatt & Zaveri, 
2001, pp. 304-305). Many data warehouse and other district-level information access projects 
never provide any services beyond basic access to data. However, without the incentive of access 
to more sophisticated system features, many potential users may conclude that learning to use the 
system is not worth the effort. In the case of a school improvement model, the limited time and 
skills of SIP team members argue for a system that contains a great deal of embedded knowledge 
that can be manipulated by novice users. 

Technical Capacity 

There is no single best approach to supporting better decision making at the school level. 
There are, however, a number of approaches to data collection and analysis that might support 
the development of more sophisticated questions and a more rigorous search for alternatives. As 
discussed above, state- or district-level testing is often the only common assessment metric 
(aside from grades) that is available for analysis of student outcomes. In addition, schools may 
have access to attendance, discipline, and other administrative data at some level of aggregation. 
In the case of elementary schools, in particular, much of this data is kept in paper form in teacher 
grade books and lesson plans. In such cases, only aggregated data is passed up to the school and 
district level for reporting student outcomes and providing compliance evidence. 

Centralized classroom record keeping is possible in some environments. The majority of 
school classrooms have Internet-connected computers and would, therefore, potentially have 
access to remote systems for entering classroom data and receiving reports. According to a 
recent National Center for Education Statistics study (Cattagni & Farris 2001), the availability of 
networked computers in classrooms has increased substantially: 

Since 1994, when 3 percent of instructional rooms had computers with Internet access, public 
schools have made consistent progress in this area: in fall 2000, 77 percent of instructional rooms 
were connected to the Internet, up from 64 percent in 1999. However, in 2000, as in previous 
years, there were differences in Internet access in instructional rooms by school characteristics. 
For example, in schools with the highest concentration of students in poverty (75 percent or more 
students eligible for free or reduced-price school lunch), a smaller percentage of instructional 
rooms were connected to the Internet (60 percent) than in schools with lower concentrations of 
poverty (77 to 82 percent of instructional rooms), (p. 1) 

Although schools and students in urban districts continue be relatively disadvantaged in 
their access to Internet technologies in the classroom, there have been substantial improvements 
in technical infrastructure. To cite a regional example, both the Madison and Milwaukee public 
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school systems have made major improvements in network infrastructure and have increased the 
number of networked computers available for administrative and instructional uses. 
Unfortunately, technology by itself is not sufficient to support robust data-based decision 
making. 



Organizational Capacity 

Technology in the classroom is important, but more important is an expanded vision of 
what constitutes data for school-level staff. As mentioned above, most district information 
systems can provide annual test score data, grades, demographics, attendance, and discipline 
data. This is the sort of information required to run the business of a district. Much of this data is 
also required by state and federal agencies that fund or regulate different segments of K-12 
activity. What is missing from this picture is data that is focused on the work of managing a 
building in support of teaching and student learning. 

One recent article (Barnes & Miller, 2001) described an innovative approach to data 
collection undertaken by the Palisades School District in Pennsylvania. Short, one-on-one 
interviews are conducted twice a year with over half of the students in the district by parents, 
teachers, and administrators (including the superintendent and assistant superintendent, who have 
each conducted over 100 interviews). This data is collected to provide an additional perspective 
on district progress toward its learning goals. 6 Teachers use the feedback they get from students 
to reflect on instructional practices, refine teaching approaches, and aid them in selecting 
curricular materials. An interview team visits a single school and spends the day doing 
interviews. At the end of the day, the team meets with the principal and the teaching staff to 
summarize the comments made by the students. The interview notes are left at the school to 
provide teachers with the opportunity to engage in a more detailed analysis of the responses. The 
survey was designed to explore and make clear the district’s expectations for particular areas of 
the curriculum each year. In this way, the administration of the survey both reinforces the 
district’s goals and gathers detailed information about students’ progress toward meeting those 
goals. The fact that the walk-throughs are conducted twice each year allows teachers and 
administrators to observe change more clearly and provides a longitudinal view of student 
experiences that is often lost as students progress from grade to grade with little interaction 
between grade cohorts. 

Active, focused, local school reform is made even more pressing and relevant by a recent 
study based on National Assessment of Educational Progress 1996 test data that showed that the 
impact of teacher classroom practice is as important as socioeconomic background and ethnicity 
(Wenglinsky, 2002). Educational improvement planning is happening at the school level and has 
the potential to have a large positive impact on student learning. One of the things that is missing 
is an open, reflective, decision-making process that would support school administrators and 
local teaching staff in their desire to engage in meaningful, locally led improvement. One of the 
biggest stumbling blocks in creating an effective environment for decision making is the lack of 
any relevant data about the actual practice of teaching. Linking larger educational goals to local 
school- and classroom-level practices must be a vital part of any serious improvement project. 



6 



The 15-minute interviews are based on a structured protocol that is adjusted to be grade-appropriate. 
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Conclusions 

There is a large literature on decision making. The Handbook of Social Psychology 
provides an excellent cross-disciplinary overview and describes structure versus process, riskless 
versus risky choices, and normative versus descriptive models of choice as the important 
dimensions of decision making (Abelson & Levi, 1985, p. 233). The authors suggest several 
approaches that would improve the quality of decision making. A number of their 
recommendations would be particularly relevant for the case study discussed here (pp. 274-293): 

1. Provide improved access to data. Improved access to relevant data — whether locally held or 
extracted from a district data store — would reduce the SIP team’s uncertainty about the 
important dimensions of the problem at hand. This clearer view of the gap in expected 
outcomes would allow the team to craft a more targeted and appropriate response. 

2. Provide access to high-quality professional development. Access to high-quality professional 
development materials in the form of case studies linking practice, curricular materials, and 
outcomes would help the team expand its search for alternatives outside the narrow box of 
improving a particular score. 

3. Review incentives that reinforce short-term gains over long-term structural improvements. 
The current climate in many urban schools is one of “improve this year or else.” Value-added 
analyses that relate performance to the rate and/or amount of improvement for particular 
groups of students should be explored as a method of rewarding movement toward a goal and 
providing incentives for improvement to both high- and low-performing schools. 

4. Expand range of approaches to school improvement planning. District school improvement 
planners need to recognize that providing school leaders with highly constrained choices 
about how to show improvement reduces these leaders’ ability to think creatively and search 
for alternatives that would be most meaningful for their particular conditions. 

State- and district-level administrators need to provide leadership by showing that they 
value appropriate use of data to support school improvement. They also need to provide 
professional development opportunities and personal mentoring necessary to make their staff 
comfortable with the process of needs analysis and program evaluation at the school level. 
Although political accountability and its related metrics are important, they are removed from 
instructional practices and provide very little relevant feedback for local, classroom-level 
improvements. 
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